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ABSTRACT 

A general issue related to language program 
development involves the empirical rationalization of cut score 
decisions in criterion-referenced language tests. Cut score 
dependability focuses on the consistency of the decisions in repeated 
testing or the assessment of language learner performances. In this 
case, the issue is to determine the optimal index of gain score 
dependability in the pre-instruction and post-instruction approach to 
assessing the language learning gains* This paper examines an 
approach used in assessing gain score dependability in which the 
optimal index of gain score dependability is derived from examining 
the cut score dependability of the pre-instructional administration 
of the criterion-referenced test as well as the post-instructional 
criterion-referenced test, in relation to differences in the ratio of 
pre- and post-instruction variances. The database comes from a 
pre-instruct ion administration of a university-level academic 
listening test followed by a counterbalanced post-instruction 
administration of an alternate form of the same test after one 
semester of instruction. Results indicated only moderate gain score 
dependability, because the cut score was close to the mean proportion 
correct on the post-instructional test. (MDM) 
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Abstract 



Muck of the recent work on criterion-referenced language testing 
addresses the issues of item writing and cut score dependability. 
Criterion-referenced item writing is centrally concerned with 
determining the content congruence and learnability of each item's 
content. Cut score dependability focuses on the consistency of 
decisions in repeated testing or the assessment of language learner 
performances. A more general issue related to language program 
development also involves empirical rationalization of cut score 
decisions. In this case the issue is of determining the optimal 
index of gain score dependability in the pre-instruction and post- 
instruction approach to assessing the language learning gains. The 
present paper examines a coiamonly used approach to assessing gain 
score dependability. The optimal index of gain score dependability 
is derived from examining the cut score dependability of the pre- 
instructional administration of the criterion-referenced test as 
well as the post- instructional criterion-referenced test, in 
relation to differences in the ratio of pre and post instruction 
variances. The database for the present paper comes from a pre- 
insvruction administration of an academic listening test followed 
by a counterbalanced post-instruction administration of an 
alternate form of the same test after one semester of instruction. 
The subjects were 213 advanced ESL learners at a large American 
university English language institute. 
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CRT in Language Testing 

The need for making language tests optimally useful for the 
assessment of second language instructional programs has been a 
point of discussion for more than a decade (Cziko 1981; Henning 
1982) . A trend toward designing tests to assess the effects of 
instruction has recently gathered momentum in the field of language 
testing. The advantages of using criterion-referenced tests in 
language programs stem from their better fit to the content of 
tasks, objectives, and linguistic structure included in second 
language syllabus, and their potential to more accurately indicate 
changes in proficiency as a direct result of instruction. 
Criterion-referenced tests also potentially provide a more 
dependable basis for program evaluation (Brown, 1991) . 

Recent discussions of the advantages of criterion-referenced 
testing in the field of language testing have tended to dwell on 
test building procedures (Brown, 1991) . A key notion in 
criterion-referenced test making is the difference index, which is 
used to assess an individual item's capacity to reflect learners' 
gain in skill or knowledge (Hudson and Lynch, 1984; Hudson, 1993). 
The difference index for an individual item is the percentage 
correct on the pre-instruction administration of the test 
subtracted from that item's post-instruction percentage correct for 
the same group of learners. 

The advantages of building criterion-referenced tests in 
program development and assessment are numerous. By designing the 
content of the test items to be optimally congruent with the 

2 



ERIC 



3 



instructional syllabus, the potential for iteia content validity is 
maximized (Brown, 1991) • Also, if there are demonstrable gains on 
some items, but not on others, the degree of learnability of 
subcomponents of the language teaching syllabus can be better 
examined and revised* In contrast to using standardized or 
norm-referenced measurements as post-instruction criteria for 
program evaluation, the criterion-referenced approach offers the 
advantage of detecting individual differences in change vis a vis 
the content of the syllabus. Norm-referenced tests, in contrast, 
tend to cover a much wider range of items by concurrently sampling 
larger domains of linguistic knowledge (Hudson and Lynch, 1984; 
Brown, 1991) . 

The introduction of criterion-referenced testing to the field 
of language testing has yet to date tended to dwell on the item 
making and interpretation process. There are implicit assumptions 
about the dependability and validity of total score gain as the 
direct result of instruction in the pre-instruction and 
post-instruction interpretation of difference indices for 
individual items. The direct comparison of pre- and post-test 
total score differences can lead to gain scores that may present 
problems for determining their reliability and validity (Lord, 
1963; cf. Rogosa and Willett, 1983). ^ 

Current approaches to criterion-referenced test dependability 
primarily rely on either comparisons of dichotomous judgements of 
mastery on two independent administrations of a test (Subkoviak, 
1980, 1988; Brown, 1990) , or rely on squared-error loss agreement 
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approaches (Berk, 1984) , that detect the proximity of a score to a 
criterion or cut score along a continuum (Brown, 1990) . One 
advantage of a squared-error loss agreement approach resides 
primarily in the fact that a single test administration is thought 
to be adequate for determining the dependability of the decision 
about individual scores (Brennan, 1980, 1984; Brown, 1990). 

The squared-error loss agreement approach, usually in the form 
of phi, calculated at a given cut-score (lambda), provides a 
dependability index for each of the criterion-referenced test 
administrations in the pre-instruction and post-instruction scheme 
appropriate for the assessment of instructional programs. Here 
lambda is a pre-determined standard for mastering set for both pre- 
test and post-test. Lambda can be a different proportion on each of 
the test administrations. 

Figure 1 Phi Lambda Index of CRT Dependability 

Where : 

lambda is the cut score expressed as a proportion 

k is the number of items on the test 

Xp is the mean of proportion scores 

Sp is the standard deviation of proportion scores 

The cut score dependabilities for the pre-instruction measure 
and the post-instruction measure do not provide information about 
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the extent and reliability of pre-to post test gains. For this 
reason, an elaboration of the criterion-referenced model is 
warranted - one that can address the dependability and ideally, the 
validity of instructional gains relative to the cut-scores utilized 
in the criterion-referenced approach to language testing. 

The present study addresses the issue of criterion-referenced 
gain score interpretation in light of dependability and validity 
issues. Our focus is on integrating dependability indices with 
pre-instruction and post-instruction variances on total test scores 
typically used in academic skill-building instructional programs. 
Criterion-referenced language test designers could potentially 
benefit from the experience of test analysts from other areas of 
educational measurement who have tackled the problem of linking 
gain scores, or changes in ability before and after instructional 
programs, to external criteria. The approach used in gain score 
validity analysis is to link achievement score differences with 
auxiliary criteria known to assess the same traits as those thought 
to be developed through instruction (Gupta et al, 1988) . The 
essential difference is that the analysis of gain scores for norm 
referenced tests have assumptions based on internal consistency and 
small standard errors of measurement while criterion-referenced 
tests assume skewedness on pre-and-post instruction distributions, 
and a pre-set definition of mastery. Brennan, 1984; Hudson and 
Lynch, 1984; Brown, 1991). 

For the practical implementation of criterion-referenced 
testing in intensive language programs, where there is an explicit 
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assumption that short term gains will accrue as the direct result 
of instruction, an implicit assumption is that observed gains are 
dependable and valid in relation to relevant criteria. The 
methodology for assessing the effect of instructional programs 
utilizing criterion-referenced assumptions, however, has not been 
examined extensively in the language testing literature. The 
examination of gain scores in terms of mean differences implies a 
familiar and straight forward approach to assessing instructional 
effect size. Individual pre-test and post-test scores can be simply 
collated and a matched t-test can be used to assess the observed 
mean gain in relation to the null hypothesis. An analogous 
approach that retains the familiar conception of dependability or 
reliability, expresses the observed gain after instruction in terms 
of a magnitude ranging form zero to unity. The sections below 
explicate how the dependability of gain scores can be used in a 
criterion-referenced context. 

The analysis of gain scores has been conducted in a variety of 
ways, but one approach that is relevant to criterion-referenced 
language testing is one that incorporates changes in the 
distribution of relative variances on pre-instruction and 
post-instruction measurements for the same cohort of students. 
Zimmerman and Williams (1982) and Williams, Zimmerman and 
Mazzagatti (1987) suggest an index of gain score reliability that 
incorporates the magnitude of the changes from pre-to-post-test in 
relation to changes in the ratio of pre-and post-test variances. In 
their approach, the reliability of the gain is greatest when the 
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pre-test and post-test are internally consistent and show a low 
correlation. Language testers familiar with gain score reliability 
will recognize the approach used widely in educational psychology. 
This approach is based on the internal consistency of the pre-test 
and post-test instruments, and their correlation* 



rlgurtt 2 Internal oonsistttncy and corralat ion-based gain 
raliability 



r + r " y T 



Where : 

is the internal conaistency of the pre-teat 
ryy is the internal consistency of the pos t-test 

is the product moment correlation between the two tests 

Zimmerman and Williams (1982), Rogosa and Willet (1983), and 
Williams, Zimmerman and Mazzagatti (1987) discuss modifications of 
the internal consistency and correlation based approach to gain 
reliability that are optimally sensitive to changes in score 
distributions from pre- instruct ion to post- instruct ion. They add 
ratios of standard deviation terms (theta, below) to make a product 
of internal consistency and the ratio of pre-test and post-test 
score distributions. The Zimmerman and Williams modification 
(Figure 3) makes explicit the assumption that greater variation 
among learners is expected before instruction relative to variation 
after instruction. Figure 3 shows the Zimmerman and Williams (1982) 
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modification of the internal consistency and correlation-based gain 
reliability. 



Figure 3 Gain mcovm reliability 

Gain Rel^ — ^ 

where: 

theta I is the ratio of pre-to-post-test standard deviations 

theta 2 the ratio of post-to-pre-test standard deviations 

r„ is the internal consistency of the pre-test 

ryy is the internal consistency of the post-test 

r^ is the product-moment correlation between the two tests 

In order for the gain score reliability concept to apply to 
the criterion-referenced test dependability interpretations, some 
adaptations are necessary. By replacing the internal consistency 
estimates for the pre-instruction administration of the 
criterion-referenced test with a squared-error loss agreement 
coefficient phi, fixed at a cut score lambda for each of the pre- 
and post- test administrations, the Williams, Zimmerman and 
Mazzagatti (1987) approach can be adapted to assess gain score 
dependability. Here, pre-instruction criterion- referenced measures 
are used as a baseline for language learning gains as indicated on 
post- instructional criterion-referenced measures for the same 
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cohort of learners • This approach is premised on there being a cut 
score on both the pre-instruction and post-instruction versions of 
the criterion-referenced tests* Figure 3 shows the modification of 
the norm-referenced approach to gain score reliability to suit the 
conditions of criterion-referenced gain score dependability • 



Figure 4 Gain 8cor« Dependability 

Gain Dependability- 



xy 



Where : 

theta , is the ratio of pre-to-post-test standard deviations 
thota 2 the ratio of post-to-pre-test standard deviations 
phi (lambda ^) is Uhe squared-error loss agreement on the pre-test 
phi (lambda y) is the aquared-error loss agreement on the post-test 
r^ is the product-moment correlation between the two tests 

Table 1-3 j^how gain score dependability for criterion- 
referenced tests. Dependability is calculated for pre-test with 
post-test correlations at ,1, .3, .5, .7 and .9 and differing cut 
score dependabilities for the pre-test and post-test 
administrations of the criterion-referenced test. 
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A Critttrion-Rof ttrancad Example 
Materials 

The criterion-referenced test data used in this study came 
from an advanced academic listening comprehension course at a large 
American university English language institute. The content of the 
criterion-referenced test passages and items matched the syllabus 
specifications for the advanced course. The test content covered 
listening skills deemed essential for advanced level English as a 
Second Language students and came from a needs analysis of critical 
listening tasks. The 21-item test consisted of six major listening 
tasks: 1) Linking referring pronouns to full noun phrases, 2) 
Recognizing cohesive devices, 3) Recognizing supporting factual 
detail, 4) Determining cause and effect, 5) Comprehending 
vocabulary in context, and 6) Note taking. 

Two parallel forms of the advanced listening test were 
developed for use as pre-tests and post-tests. Each form of the 
test consisted of thirteen short lecturettes delivered in a 
narrative style. Each form of the test took approximately twenty- 
six minutes to complete (excluding instructions) . The short 
criterion-referenced tests (k=21) were designed to assess specific 
objectives of the course. Their length reflects the assumption that 
domain- specific tests can sample skill areas as efficiently as 
longer assessments (Hudson and Lynch, 1984; Brown, 1991). 

The pre-test form was administered in the first week of 
instruction. The strategy for using pre-test measures was 
predicated on the assumption that students surpassing the pre-test 
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cut score before instruction got underway could be exempted from 
the course* The post-test version was the alternate form. The two 
forms were periodically switched so that each form would be used as 
pre-test and post-test in different academic terms* The second test 
administration came at the end of approximately forty hours of 
classroom instruction in advanced listening skills. 

Different criteria for determining passing scores were used on 
the two administrations of the test. On the pre-test, the cut- 
score was set at 90% correct of the twenty-one item test. The cut- 
score for the post-test was set considerably lower at 60% correct. 

Sxibjttots 

The test data used in this study were gathered on the total 
pretest and post-test scores of 213 matriculated students enrolled 
in the advanced listening course. Test records for the 213 subjects 
were selected from an archive of test results on the condition that 
both the pre-test and post-test had been completed. No subjects had 
missing test scores and were enrolled in the advanced academic 
listening course during of three semesters (Fall 1991^ Spring 1992 ^ 
Fall 1992) . All students enrolled in the advanced listening course 
were either placed directly into the level by a multi-passage 
academic listening test and dictation used for placement into a 
two-level academic listening program, or were promoted from an 
intermediate-level listening course taken in the previous academic 
term* 
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Analysis 

The assessment of gain was based on the students' total scores 
on the pre- instruct ion and the post-instruction tests ♦ The most 
direct indication of gain is the difference between pre-test and 
post-test scores ♦ Each pair of scores is independents That is, no 
student took the pre-test or post-test more than once* A paired-t 
test was calculated to examine the significance of the differences 
between students' performance on the pretest and the post-test ♦ The 
squared-error loss agreement dependability index was then computed 
to detect the dependability of the cut score (90% on the pre; 60% 
on the post) in determining masters from non-masters ♦ Finally, 
Williams, Zimmerman and Maz^agatti's (1987) approach to gain score 
reliability was modified by replacing the internal consistency 
estimates of pre-test and post-test with the squared-error 
agreement coefficient in order for it to apply to the criterion- 
referenced test scheme* 

Williams, Zimmerman and Mazzagatti (1987) provide three 
approaches to gain score dependability ♦ All three were examined in 
the present study using data from the criterion-referenced test 
administration. The present discussion will be limited to the 
simple gain score approach outlined by Williams, Zimmerman and 
Mazzagatti. The simple gain is dependent on the internal 
consistency of the pre-test scores. Since in this analysis we are 
dealing with criterion-referenced assumptions, we replace the pre- 
test and post-test internal consistency estimates with phi (lambda) 
estimates* 



Gain Dapandabillty Results 

The modification of the Williams, Zimmerman and Mazzagatti 
(1987) approach provides the basis for determining the 
dependability of the gain observed in the advanced listening 
course. As can be seen in Table 4 there was greater variance on 
the pre-test relative to the post-test, indicating that the 
advanced language learners were more homogeneous after instruction. 
Mean scores on the two tests were also significantly different 
(paired t= -6.55, p<.005). 

Tabls 4 

Xjgj=0.57 0 Pre-test mean proportion 0.648 Post- test mean prop 

5^=0 . 16 Pre-test s. d. proportion 3yp=0 . 12 Post-test s. d. prop 
Xj^=0.90 Pre-test cut score X^=0.6 0 Post- test cut score 

k^=21 Pre-test items ky=21 Post- test items 



Since the cut scores on the two administrations were 
different, we observed considerable variation in the dependability 
of the two criterion-referenced tests. On the pre-instructional 
administration, where the cut score was set at 90%, the median 
score of the advanced language learners was well below the 
criterion, resulting in a high pre-test dependability (Figure 5, 
below) (phi-jc) . 
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Figure 5 Pr«-ttt8t Dependability 



=0,92 



xp 



The post-instruction test, in contrast, resulted in a mean 
proportion score very close to the 60% threshold for defining a 
passing score. The post-instruction dependability (phi-y) (Figure 
6, below) therefore reflects the decreased dependability associated 
with making decisions about scores so close to the criterion score 
for mastery of course content. 

Figure 6 Post-test Dependability 

The gain score dependability index (GDI) (Figure 7) reflects 
modest dependability of gain on this post-instructional criterion- 
referenced test. The small decision dependability on the post-test 
qualifies the interpretation that the gain on the pre-post 
comparison was uniform among the advanced language learners in the 
course. 



= 0,37 



Figure 7 Gain Score Dependability 
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It is also contingent on the correlation (see Table 5) between 
the pre-instruction and post-instruction forms of the criterion- 
referenced tests. In circumstances where we would expect learners 
to have no knowledge of the trait, the pre-post correlation should 
be near zero. Gain score dependability is largest when the 
component criterion-referenced measures are dependable and are 
based on appropriate cut scores, and when the pre-post correlation 
is near zero. 



r =0 ,23 Pre-'Post Correlation 



For the present advanced listening course data set, we find 
only moderate gain score dependability, because the cut score is 
close to the mean proportion correct on the post-instructional 
test. 

Gain 8cor# Validity 

The linking of internally valid instructional effects in 
program development is not particularly new in language testing. 
Henning (1982; 1988) devised approaches to assessing growth- 
referenced evaluation for language programs that depend primarily 
on internal ly-based definitions of validity. The approacli followed 
here anchors external criteria to the pre-to-*post gains. Gain 



Table 5 




1.28 



Q^^_£03t =0.78 Ratio of CRT variances 
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7lgur# 7 thm distribution of pre- and post-test 

score validity is premised on the logic that observed gains are 
relatable to some other external criterion - provided that the 
external criterion measures the same latent trait as that reflected 
in the gain scores. Validity is largest when the pre-test and 
external criterion correlation is zero and the post-test with 
criterion correlation is very high. We would presume, for example, 
that before instruction learners' performance would show no 
correlation with criterion measures . Learner performance after 
instructional goals have been dependably achieved will reflect the 
skill or knowledge that can be correlated with an external 
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criterion assessing the same domain of knowledge or skill. Before 
instruction, we do not assume that there is any basis for such a 
correlation. Gupta, Srivastava and Sharma (1988; 1989) define 
gain score validity as one based on relative magnitudes of 
pre-post-external criterion correlations. 



Figure 8 Gain Score Validity 



where : 

ia the product -moment correlation between the post-teat and the external 
criterion* 

r„ ia the product-moment correlation between the pre-test and the external 
criterion* 

r^ ia the product-moment correlation between the pre-test and the post-test 

The approach adopted here includes the gain score 
dependability index so as to make the validity index optimally 
conditioned on the dependability of the pre-to-post instructional 
gains. We therefore modified the Gupta et al gain score validity 
approach to suit the conditions of the criterion-referenced 
approach in the present study, although no external criterion was 
available for the validation of the observed gains. The exposition 
below is therefore meant to demonstrate how a gain score validity 
component can be extended from the gain score dependability 
approach thus far discussed. 
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Figure 9 Extended Gain Score Validity 



whttr«: 

is the pre-instruction with post-instruction CRT correlation 
r„ is the pre-inatruction with external criterion correlation 
ry, is the post-instruction with external criterion correlation 
GDI is the gain score dependability index 

Ideally, the external criterion would be a parallel form of 
the criterion-referenced test in a narrowly specified domain 
related to instructional objectives. The external criterion could 
itself be validated through conventional means such as multi-trait 
multimethod approaches used in language testing (Stevenson, 1980; 
Bachman and Palmer, 1982; Henning and Dandonoli, 1991). 

Implioations for criterion-referenced language testing 

With increasing use of criterion-referenced tests in the 
evaluation of language teaching programs, there is a concurrent 
need for determining the extent to which gains can be reliably and 
validly related to important criteria. The match between the 
content of instructional programs and the observable gains is 
ideally related to criteria grounded in the needs of language 
learners in institutions. The role of external criteria in the 
validation of the instruction gains is therefore essential for 
the development of a dynamic language teaching program, especially 
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in an academic context. The use of external criteria for gain score 
validity assessment is based on a number of constraints. 

In order for there to be optimal assessment of gain in a 
language teaching program, there are strong assumptions about the 
degree of variance overlap between pre-instruction and post- 
instruction measures. In order for gain to be assessed more 
clearly, the correlation between the pre-to-post instruction 
measures should be approximately zero. This is a severe assumption 
for most language teaching programs because academic second 
language learners initially matriculate with a high degree of 
proficiency. Since language skills tend to be robustly 
intercorrelated, the potential for finding specific linguistic 
subskills that are readily identifiable and teachable also presents 
a constraint on ascertaining gain score validity. 

The gain score validity index is dependent on the external 
criterion, the pre-to-post instruction correlations and the gain 
score dependability index. The gain score dependability index is 
itself subject to the pre-determined cut scores. The basis of the 
cut scores in criterion-referenced testing (Messick, 1988) is 
notoriously difficult to justify in absolute terms. Knowing "how 
much is enough" is typically beyond the standard setters to agree 
on. In the present study, the 60% on the post-instruction test 
reflects the intention to make a commonly used threshold in 
academic settings the minimum criterion for passing. The pre-test 
cut score of 90% is perhaps less justified. Its function is mainly 
to allow learners who have been misplaced to demonstrate clear 



mastery of the course content, and therefore make themselves 
candidates for exemption from the required course of instruction. 
In order to make criterion-referenced tests optimally dependable 
and equitable to language learners, the pre-instruction cut score 
should be based on a realistic criterion. One such candidate could 
be determined by the average proportion of answers correctly 
answered by several previous cohorts of instructed learners. The 
pre-instruction cut score could be rationally defined on the 
average performance of persons who have previously mastered the 
course content. 

Making Criterion-Ref aranoed Tests Work 

The crucial element in making criterion-referenced language 
tests work is the interface of the instructional syllabus and the 
domain of language knowledge to be taught and assessed. The ideal 
criterion-referenced test, one that leads to clearly observable 
gains, is one that samples a knowledge domain that learners do not 
already possess. This ideal is reflected in the assumptions 
underlying the gain score validity index - one of which is that 
pre-instruction and post-instruction correlations are near zero. 
This assumption, however, will no doubt be extremely difficult to 
satisfy in instruction programs that focus exclusively structural 
language teaching and testing - on curricula designed to cover only 
linguistic knowledge. Assuming that matriculated university English 
as a second language students already possess advanced knowledge of 
the language, developing criterion-referenced tests is especially 



difficult, and px^esents a potential criterion-referenced dilemma 
for language testers. The more discrete point the teaching 
syllabus and the test content become, the more observable the gains 
will most likely be. Whether the gains accruing from a narrowly 
defined dciuain come at the expense of content validity, and the 
development of crucial academic skills can only be determined by 
the contiviual analysis of the interface of criterion-referenced 
test content, learner needB and external criteria. 

An alternative to the systemic language teaching and testing 
approach now prominent in academic language programs is one that 
integrates procedural knowledge of language-dependent academic 
research skills into both the teaching syllabus and criterion- 
referenced testing scheme. An example of such and approach would 
integrate the teaching of advanced research techniques into a task- 
based api^roacb to criterion-referenced test design. In this 
scheme, the content of criterion-referenced tests is not 
exjilusively focused on language structure, reading skills, 
vocabulary expansion so much as it extends advanced learners' 
knowledge of specific research procedures in a modern research 
library* The content of these criterion-referenced tests includes 
procedural knowledge of advanced reading tasks , e.g., tasks 
simulating compact disk ROM database search procedures. In contrast 
to purely systemic language pre- instruction tests, which typically 
show a high mean and little variance, the integration of procedural 
research tasks with systemic knowledge tests better fits the low 
pre- instruct ion mean score assumptions of currently conceived 



criterion-ref erencad tests. The task-based approach to criterion- 
referenced testing thus shows potential for accomplishing two 
important goals. One, to show that the effects of intensive 
instruction in academic preparation programs can result in tangible 
gains, and two, that the content of criterion-reference tests can 
be integrated in academic task simulations that serve to provide 
comprehensive review of systemic language knowledge while at the 
same time provide practice in crucial academic research skills for 
advanced learners. 
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